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Harvard Family Research Project’s series of Out-of-School Time Evaluation Snapshots distills the 
wealth of information compiled in our Out-of-School Time Program Evaluation Database 1 into a 
single report. Each Snapshot examines a specific aspect of out-of-school time (OST) evaluation. 
This Snapshot provides the academic, youth development, and prevention performance measures 
currently being used by out-of-school time programs to assess their progress. It also includes the 
corresponding data sources for these measures. 2 A future Snapshot will focus on the specific 
standardized assessment tools and tests that programs use to demonstrate impact. 3 

Increasingly, out-of-school time programs are being asked to provide data that documents their 
progress and demonstrates their results. But for many OST programs and their over-worked staff 
this added responsibility is the “straw that broke the camel’s back.” Across the country there is a 
cry from programs for help in deciding what to measure and how to measure it. Although there is 
no formal consensus regarding realistic outcomes for all OST programs, one of the unintended 
benefits of the release of the first national 21 st Century Community Learning Centers impact 
evaluation 4 has been increased attention to the question “What is realistic for OST programs to be 
held accountable for?” Proceedings from a June 2003 After School Summit hosted by the U.S. 
Department of Education and Arnold Schwarzenegger indicate that researchers and policymakers 
alike are interested and invested in helping programs measure and improve their performance. 5 

This Snapshot contributes to that conversation by providing a comprehensive listing of the 
performance measures that OST programs nationwide are currently using as well as the sources 
that they use to collect data on each measure. This Snapshot is based on a review of the outcomes 
reported in the evaluations posted in the Harvard Family Research Project Out-of-School Time 
Program Evaluation Database. 6 While not an exhaustive listing of the universe of performance 
measures currently being used, this Snapshot provides information for programs making evaluation 
decisions as well as for researchers, evaluators, and others helping programs build evaluation 
capacity and improve performance. 



1 Our database contains profiles of out-of-school time (OST) program evaluations, which are searchable on a wide range 
of criteria. It is available in the OST section of the HFRP website at www.gse.harvard.edu/hfrp/projects/afterschool/ 
evaldatabase.html. 

2 Performance measure information for this Snapshot was adapted from Documenting Progress and Demonstrating 
Results: Evaluating Local Out-of-School Time Programs, a joint publication of the Harvard Family Research Project and 
the Finance Project, available at www.gse.harvard.edu/hfrp/projects/afterschool/resources/index.html#local. 

3 This and future Snapshots in the series will be available in the OST section of the HFRP website at 
www.gse.harvard.edu/hfrp/projects/afterschool/resources/index.html. (To be notified when Snapshots become available 
sign up for our OST website change notification email at www.gse.harvard.edu/hfrp/subscribe.html.) 

4 U.S. Department of Education, Office of the Under Secretary. (2003). When schools stay open late: The national 
evaluation of the 21st-Century Community Learning Centers program, first year findings. Washington, DC: Author. 
Available at www.ed.gov/pubs/21cent/firstyear. 

5 A summary report of the After School Summit is available at www.publicengagement.com/afterschoolsummit. 

6 Our scan for this Snapshot was conducted in October 2003, at which time there were 54 profile posted in our database. 
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What Is a Performance Measure? 

Performance measures assess your program’ s progress on the implementation of your strategies 
and activities. They assess the results of your out-of-school time (OST) program’s service 
delivery. Ask yourself: In the work that my program does, what do we hope to directly affect? 
What results are we willing to be directly accountable for producing? What can we realistically 
accomplish? 

There are two types of performance measures: 

Measures of effort - Also commonly known as outputs, these are measures of the 
products and services generated by program strategies and activities. Ask yourself: 

What does my program generate (e.g., publications, training materials). What levels of 
activity do we produce (e.g., the number of children served or products developed), and 
What will measure the quality of our services (e.g., parent and child satisfaction rates)? 
Measures of effort assess how much you did, but do little in terms of explaining how 
well you did it or how well your program ultimately worked for the target population 
with whom you are working. These arc the easiest of all the evaluation measures to 
identify and track (e.g., number of children served in the OST program and participant 
demographics, number of classes/sessions/trainings held, etc.). 

Measures of effect - These arc changes in knowledge, skills, attitudes, or behaviors in 
your target population. Ask yourself: How will I know that the children or families I 
work with in my OST program arc better off? What changes do I expect to result from 
the strategies and activities my program provides? Remember that measures of effect 
reflect changes that your program acting alone expects to produce (e.g., increased social 
competence, higher self-esteem and confidence, improved study habits). 

Adapted from: Watson, S. (2000). Using results to improve the lives of children and families: A guide for public- 
private child care partnerships. Vienna. VA: Child Care Partnership Project. Available at nccic.org/ 
ccpartnerships/results.pdf (Acrobat file) and Coffman, J. (2002). Learning from logic models in out-of-school time. 
Cambridge, MA: Harvard Family Research Project. Available at www.gse.harvard.edu/hfrp/projects/afterschool/ 
resources/learning_logic_models.html. 



Performance Measures and Data Sources for Academic Outcomes 

Evaluators use a broad range of performance measures to assess participants’ academic outcomes 
in OST programs. They range from grades to standardized testing to participants’ homework 
completion. The list below shows the broad range of possibilities for OST academic performance 
measures. Obviously a single program would not use all these measures and may develop some not 
on this list. Data sources for these performance measures include: parent, teacher, principal, OST 
staff, and participant interviews and surveys; school records; and standardized testing results. 

Data Sources for Academic Performance Measures 

Performance Measure Data Source(s) 

Ability to get along with others in school Parent 

Academic knowledge in specific content areas Participant 

Academic performance in general Participant, parent, principal, program staff 
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Attendance/absenteeism 

Attitude toward school 
Behavior in school 7 

Communication skills in school 
Cognitive improvement 
College attendance 

College preparation/plans/information 
College retention/continuation 
Comfort with taking tests 
Disciplinary action/suspension/expulsion 
Dropout 

Effectiveness of school overall 
Effort (including grades for effort) 

English language development 

Expectations of achievement and success 

Family involvement in school events 

General academic knowledge 

Grade point average 

Grade retention 

Grades - overall 

Grades in content areas (math, reading, etc.) 
Grades in OST program academic courses 
Help seeking 
Homework performance 
Intention to graduate 
Intention to remain in school 
Interest in recreational reading 
Learning skills development 
Liking school 
Motivation to learn 
Overall happiness in school 
Perceived competence 
Quality of schoolwork 
Recreational reading 
Safety - viewing school as a safe place 
Scholastic achievement assessed by 
knowledge about specific subjects 
Scholastic competence 
School vandalism 
Special education placement 



Participant, school records, teacher, parent, 

principal 

Participant 

Principal, teacher, standardized behavior 

scales 

Parent 

Teacher 

Participant youth survey, school records 

Participant 

Participant 

Participant 

School records 

Participant 

Principal 

Participant, school records 

Participant 

Participant, teacher 

Principal 

Participant 

School records 

Participant, principal, school records 

Participant, teacher 

School records, parent 

Participant 

Participant 

Participant, teacher, parent, principal 

Participant 

Participant 

Program staff 

Teacher 

Participant, parent 

Participant, principal 

Parent 

Participant 

Participant 

Parent, teacher 

Parent 

Standardized behavior scales 

Participant 

Principal 

Principal 



7 School behaviors included in the scales are: frustration, tolerance, distraction, ignoring teasing, nervousness, sadness, 
aggression, acting out, shyness, and anxiety. 
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Standardized test scores 



SAT-9, PSAT, Comprehensive Test of Basic 
Skills, Stanford Diagnostic Reading Test 
(SDRT-IV), Terra Nova Reading/Language 
Arts and Mathematics Computation Test, 
CTB/McGraw-Hill CAT-5 Math and Reading 
Comprehension Tests, Informal Reading 
Inventory, state assessments, school-wide 
assessments, program-specific tests 
Study skills Participant 

Performance Measures and Data Sources for Positive Youth Development Outcomes 

Many OST evaluations assess youth development outcomes, which are broadly defined as those 
outcomes that assess the social and emotional development of program participants. Outcomes that 
fall into this category range from standardized measures of self-esteem, participant behavior, and 
interpersonal skills to decision making and goal setting, leadership, and career development. The 
list below shows the broad range of possibilities for OST youth development performance 
measures. Obviously a single program would not use all these measures and may develop some not 
on this list. Like the data sources for academic performance measures, sources for youth 
development performance measures include: parent, teacher, principal, OST staff, and participant 
interviews and surveys; school records; and standardized assessments. 



Data Sources for Youth Development Performance Measures 



Performance Measure 

Adult relationships 

Awareness of community resources 

Banking skills/financial skills 

Behavior change toward new program component 

Communication skills 

Computer skills 

Conflict resolution 

Cultural awareness 

Depression prevention 

Developmental Assets 8 

Exposure to new activities 

Facing issues outside of OST program 

Independence 

Interaction with adults 

Interaction with other students in OST 

Interaction with peers 

Interest in nonacademic subjects (art, music, etc.) 
Leadership development/opportunities 
Life skills 

Opportunities to volunteer 
Peer relationships 
Problem-solving skills 
Productive use of leisure time 



Data Source(s) 

Participant 

Participant 

Participant 

Parent, participant, program staff 

Participant 

Participant 

Program staff 

Participant 

Parent, teacher 

Participant youth survey 

Program staff, principal 

Participant 

Participant 

Teacher, principal, program staff 
Participant 

Participant, parent, program staff 
Participant, program staff 
Participant, program staff 
Participant youth survey 
Participant 

Standardized assessment, program staff 

Participant 

Participant 



8 Search Institute 40 Developmental Assets. For more information on this assessment framework, see www.search- 
institute.org/assets. 
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Professional/workforce skills and development 

Respect for others 

Self-confidence 

Self-esteem 

Sense of belonging 
Sense of community 
Sense of safety 
Social skills 

Sources of support for youth 



Participant 

Participant 

Parent, provider, standardized tests, employer 

Participant youth survey, standardized 

assessment, teacher 

Participant 

Participant 

Participant 

Parent 

Participant 



Performance Measures and Data Sources for Prevention Outcomes 

Performance measures that fall into the prevention category include participants’ changes in sexual 
behavior, feelings of personal safety, changes in drug and alcohol use and abuse, and overall 
improvements in physical health. The list below shows the broad range of possibilities for OST 
prevention performance measures. Obviously a single program would not use all these measures 
and may develop some not on this list. Data sources for this information include: parent, 
participant, mentor, and teacher interviews and surveys and police reporting and records. 

Data Sources for Prevention Performance Measures 



Performance Measure 

Aggression 

Birth control use 

Crime rate - youth perpetrators 

Crime rate - youth victims 

Delinquency 

Delinquency - attitudes about it 
Delinquent peer group membership 
Gang membership 
“Kept out of trouble” 

Physical fitness 
Pregnancy and birth rates 
Race relations 
“Risk taking” 

Sex - attitudes about it 
Sexual behavior/initiation of intercourse 
Sexual decision making 
Sexual knowledge 
Substance use - attitudes about it 
Substance use decision-making skills 
Substance use/drug use/alcohol use 
Weapons possession/use 



Data Source(s) 

Participant, mentor 

Participant 

Police records 

Police records 

Participant, mentor 

Participant 

Participant, mentor 

Participant, mentor 

Participant 

Parent 

Participant 

Participant 

Parent, teacher 

Participant 

Participant 

Participant 

Participant 

Participant 

Participant 

Participant, mentor 

Participant, mentor 



Which Performance Measures Are Best? 

There may never be a single set of performance measures that all OST programs can be held 
accountable for, but there are at least four important points that all OST programs must take into 
consideration when selecting performance measures for evaluation: 
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1. The range of performance measures currently used to assess program OST outcomes reflects 
the diversity of OST programming. The selection of which performance measures arc best 
suited to any single program or initiative should be inextricably tied to the program’s goals, 
strategies, and activities. Being intentional about a theory of change — a way of articulating the 
program's primary goals, strategies, and activities — can help to determine what measures to use 
to assess progress toward achieving program goals. 

2. Performance measures are not the same as performance indicators. Indicators, such as 
“increasing the percentage of students graduating from high school,” are measures for which 
data exist that quantify and track community-wide progress toward results. They require 
community-wide effort to move and reflect substantial changes across a community. The 
selection of performance measures should reflect the context in which the OST program 
operates, recognizing the strengths and limitations of the program to affect overall change 
given that context. And they should contribute to understanding progress on the program’ s 
performance indicators. 

3. Availability of data sources is a consideration when selecting and developing performance 
measures. Many programs rely on parent, participant, and staff reporting as data sources, using 
program-generated surveys and questionnaires to collect data. This is a less costly option than 
using standardized academic and behavioral assessments that may require training to 
administer, but has less validity than standardized testing and assessment tools. 

4. Performance measures should, in part, be selected because they will yield useful information 
for program improvement as well as to fulfill accountability requirements. A litmus test for a 
good evaluation, and consequently the list of performance measures selected, is to ask the 
question, “Will the information collected be useful to the program and its stakeholders?” The 
answer should be a resounding “yes.” 

Priscilla Little, Project Manager; Erin Harris, Research Assistant; Suzanne Bouffard, 
HFRP Consultant 
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Harvard Family Research Project Out-of-School Time Program Evaluation Database 

The Harvard Family Research Project (HFRP) Out-of-School Time Program Evaluation 
Database contains profiles of out-of-school time (OST) program evaluations. Its purpose is to 
provide accessible information about previous and current evaluations to support the 
development of high quality evaluations and programs in the OST field. 

Types of Programs Included in the Database 

Evaluations in the database meet the following three criteria: 

1 . The evaluated program or initiative operates during out-of-school time. 

2. The evaluations aim to answer a specific evaluation question or set of questions about a 
specific program or initiative. 

3. The evaluated program or initiative serves children between the ages of 5 and 19. 

Types of Information Included in the Database 

Each profile contains detailed information about the evaluations, as well as an overview of the 
OST program or initiative itself. Electronic links to actual evaluation reports, where available, 
are also provided, as arc program and evaluation contacts. 

How to Use the Database 

The database is located in the OST section of the HFRP website at www.gsc.harvard.edu/hfrp/ 
projects/afterschool/evaldatabase.html. The search mechanism allows users to refine their scan 
of the profiles to specific program and evaluation characteristics and findings information. 



© 2004 President & Fellows of Harvard College. All rights reserved. This publication may not be reproduced in any way 
without written permission from Harvard Family Research Project. 



About Harvard Family Research Project (HFRP) 

Founded in 1983 by Dr. Heather Weiss, HFRP conducts research about programs and policies that 
serve children and families throughout the United States. By publishing and disseminating its 
research widely, HFRP plays a vital role in examining and encouraging programs and policies that 
enable families and communities to help children reach their potential. 
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